Multimedia messaging method and system

ABSTRACT

A system for and method of controlling retransmission of a content item contained in a multimedia message, comprising receiving the message containing the content item from a sender together with an identifier of an intended recipient of the message, obtaining an identifier for the content item, retrieving approval information from a provider based on the identifier for the content item, and controlling retransmission of the message including the content item to the intended recipient based on the retrieved approval information. Controlling may comprise e.g. charging a fee for the retransmission, restricting forwarding of the content item or disallowing the retransmission. If the additional information identifies plural providers, retransmission can be made conditional upon approval by any of the plural providers.

The invention relates to a method of controlling retransmission of acontent item contained in a multimedia message. The invention furtherrelates to a system arranged for controlling retransmission of a contentitem contained in a multimedia message.

Multimedia Messaging Service (MMS) is a standard that lets users sendand receive multimedia messages with formatted text, graphics, audio andvideo clips. MMS messages can be sent to (MMS enabled) mobile phones orstandard e-mail addresses. The sending entity can for instance be amobile phone or an application running on a computer.

FIG. 1 shows the generic MMS Person-to-Person flow. A first person wantsto send an MMS message to a second person. He takes his mobile phone 101or other MMS user agent, composes the message and includes one or moremultimedia objects in the message. After hitting the “send” button, themultimedia message (text and the multimedia objects) is transmitted overa mobile network 110 to a Multimedia Messaging Service Center (MMSC)120.

The mobile network 110 could be e.g. GSM, GPRS, UMTS or a comparablesystem. As multimedia messages can potentially grow very big, abroadband mobile network is preferred. The first person could also havecomposed the message on his personal computer connected to the Internet.The message would then have been sent to the MMSC 120 over the Internet.Other ways to transmit multimedia messages are of course also possible.It is important to note that the message is always sent to the MMSC 120and not directly to the recipient. This architecture is an example of astore-and-forward messaging system.

The multimedia message is received by the MMSC 120. From there it isforwarded to the intended recipient (the second person). The multimediamessage is once more transmitted over a mobile network 130, this time tothe mobile telephone or other user agent 141 of the second person. Uponarrival of the multimedia message, this mobile telephone processes thecontent and presents the message to the second person. The two mobilenetworks 110, 130 in question may or may not be the same.

In the above example, the first person used the second person'stelephone number as the address to which the multimedia message was tobe sent. He could also have used the second person's e-mail address. Themultimedia message would then have been forwarded by the MMSC 120 to thesecond person's e-mail box, from which the second person could have readit using a standard e-mail client.

FIG. 2 shows the generic MMS Application-to-Person flow. Multimediamessaging is of course not restricted to messages sent by persons topersons. Many automated services can be offered. For instance, a popularservice is the making available of ringtones or logos for mobiletelephones. In such an application, a user can send a request for aparticular ringtone or logo to a server from his mobile telephone. Anapplication 201 on the server retrieves the appropriate ringtone or logoand transmits it as a multimedia message to the MMSC 120. Once themessage arrives at the MMSC 120, it is handled and forwarded in the sameway as described above with reference to FIG. 1. When the mobile phone141 receives the multimedia message, it extracts the ringtone or logo.

Delivering ringtones and logos is already very popular with the standardtext-based Short Messaging Service (SMS). However, MMS has a muchbroader application. A potential use of Application-to-Person MMS is theprovision of (high) value content. An example could be a service whichcan provide the latest music samples or video clips. The user pays anextra amount for that message to cover the content licensing costs, asmost of this content is copyrighted or protected by other intellectualproperty rights.

A problem occurs if both presented models are combined: The user whopaid for the content (application-to-person) can forward this message(and thus the content) to another user (person-to-person), only payingfor the transmission of the message, but not paying any license fees duefor forwarding the content.

It is an object of the invention to be able to restrict or otherwisecontrol the forwarding of multimedia content.

This object is achieved according to the invention in a method asclaimed in claim 1, and in a system as claimed in claim 10. According tothe invention the service provider who originally provided the contentitem is contacted and forwarding is conditional upon his approval. Thishas the advantage that the entity handling the forwarding does not needto keep track of policy rules regarding when it is or is not allowed toforward a content item. Furthermore, any such rules can be changed atany time by the provider without having to update anything in theforwarding system. The sytem simply contacts the provider and learnsfrom the response whether forwarding is permitted.

In an embodiment the invention uses robust fingerprints to identify thecontent transmitted in a multimedia message. The provider can beidentified using a database lookup using the fingerprint as a key. Asingle fingerprint may be associated with identifiers for pluralproviders, which means that any of these providers can approveforwarding. This makes the invention very flexible in terms of licensingoptions and enables for example price-based competition betweenproviders.

Restricting forwarding, for example by disallowing retransmission of theentire message, or by allowing retransmission of the message butdisallowing retransmission of the content item, is usually desired.Another option is to allow forwarding of the content item and trackingsuch forwarding. By tracking the content, useful information about thedistribution of the content can be gathered (for marketing purposes),but also business rules can be added to limit this distribution of thecontent.

Advantageous embodiments are set out in the dependent claims.

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments shown in the drawing, inwhich:

FIG. 1 schematically illustrates a multimedia messaging system in whichmessages from one user are passed via an MMSC to another user;

FIG. 2 schematically illustrates a multimedia messaging system in whichmessages from an application are passed via the MMSC to a user;

FIG. 3 schematically illustrates an preferred embodiment of theinvention in which a fingerprinting module is added to the MMSC;

FIG. 4 schematically illustrates an embodiment of the invention in whicha fingerprint-based identification module is added to the MMSC;

FIG. 5 schematically illustrates a Media Mediation System coupled to theMMSC; and

FIG. 6 schematically illustrates an embodiment of the MMSC.

Throughout the figures, same reference numerals indicate similar orcorresponding features. Some of the features indicated in the drawingsare typically implemented in software, and as such represent softwareentities, such as software modules or objects.

An illustration of a preferred embodiment of the invention is shown inFIG. 3. In this embodiment, the MMSC 120 is extended with afingerprinting module 310 through which all content items present inmultimedia messages must pass. The fingerprinting module 310 computes afingerprint for the content item in question. This fingerprint serves asan identifier for the content item.

One method for computing a robust fingerprint is described ininternational patent application WO 02/065782 (attorney docketPHNL010110), although of course any method for computing a fingerprintcan be used. The fingerprinting module 310 then performs a databaselookup in database 311 to determine whether this particular fingerprintis present in the database 311. If not, the fingerprint of the contentitem in question is added to the database 311 together with anidentifier for the sender of the content item. This sender then mostlikely is the original provider of the content item. If desired, anadditional check can be performed to determine whether this is truebefore adding the sender to the database 311.

Another option is to add the fingerprint of any content item to thedatabase 311 when it is first made available by the provider, not whenit is first transmitted from the provider to a client.

The MMSC 120 is also provided with a fingerprint-based identificationmodule 410, as shown in FIG. 4. The fingerprint-based identificationmodule 410 retrieves additional information from the database 311 basedon the fingerprint computed by the fingerprinting module 310. Ifadditional information can be found, this indicates that the content haspassed through an MMSC previously. This is proof that the content isbeing forwarded. As such forwarding is to be prevented or at leastrestricted, the MMSC 120 can now take appropriate action. It is notedthat if multiple MMSCs are going to be used, the database 311 should beaccessible to all MMSCs.

The simplest action is to simply delete the multimedia message ratherthan forwarding it. However, this may not be acceptable to users.Usually one part of the message (such as a picture or ringtone)represents forwarded content. Other parts, such as accompanying textmessages or pictures, are original content. The original content shouldnot be blocked simply because there is also a forwarded part in themessage. The MMSC 120 could strip out this forwarded piece of contentand pass on the remaining parts of the multimedia message. Preferably,any parts that have been stripped out are replaced by a messageindicating that they have been stripped out and why.

Alternatively, a license fee required for forwarding the content couldbe computed. This license fee can then be charged to the sending user'saccount, if this information is available. This could work well e.g.with music content, as music rights clearinghouses are available tonegotiate a general compensation or license fee.

The MMSC 120 could also convert the content to a relatively low qualityversion before forwarding the content to the intended recipient. Thisway, the intended recipient would still get the message and the content,but because of the low quality it would not be very valuable. He wouldthen be encouraged to download the original version from the MMSapplication 201 himself.

The action can also comprise reporting the distribution of this contentto a third party such as the service provider. Distribution can then bemade conditional upon approval from the provider. This embodiment worksbest if an identifier for the provider is embedded in the content item.

Another example is the allowance of the forwarding, but billing thesending user a premium price for it. In mobile networks, the sendinguser can be identified e.g. through the telephone number to allow suchbilling. Price information can be appended to the message or be suppliedin a separate message.

Appropriate action could be taken based on the retrieved additionalinformation. For example, if the additional information identifies theinitial recipient of the content being forwarded (i.e. the person whoreceived it from the MMS application server in FIG. 2), the MMSC 120could compare this identifier against an identifier for the sender ofthe intercepted multimedia message. If these two are the same, it couldpermit forwarding. This way, the initial recipient can forward contentto his friends, but those friends cannot pass the message on to anyoneelse.

Alternatively, the identifier could be used to track down the initialrecipient of the content, so that he can be charged a certain amount ofmoney for this premium service, or so that he can be blocked from usingthe MMS application. He could also be offered a subscription before heis allowed to forward content items.

If the additional information identifies the MMSC which added thefingerprint to the database 311, or the operator of one or more MMSCs,the MMSC 120 could allow forwarding of the message if it were the sameMMSC as the MMSC which added the fingerprint. This permits forwardingonly to and from users of the same MMSC.

If the additional information identifies the provider of the contentitem, retransmission can be made conditional upon approval by thatprovider. This is preferably realized by having the fingerprintingmodule 310 store the computed fingerprint associated with an identifierfor the provider in the database 311 if no identifier for the providercould be retrieved from the additional information. This way, thefingerprint-based identification module 410 can retrieve this identifierand contact the provider to obtain permission to retransmit the messageand/or the content item, or to take any other action as explained above.

If the additional information identifies plural providers,retransmission can be made conditional upon approval by any of theplural providers. The fingerprint-based identification module 410 cannow contact all of these providers and permits retransmission if atleast one of them approves. If more than one provider approves, the mostliberal approval could be applied, or the sender could be asked to pickone of the providers, or a selection could be made in some other way.

Identifying plural providers can be implemented by storing thefingerprint associated with one provider when it is found that thefingerprint is not yet in the database 311. If subsequently thefingerprinting module 310 notices that the fingerprint is in thedatabase 311 but not associated with the sender of the content item,then an identifier for this sender can be added to the entry for thatparticular fingerprint. The fingerprint-based identification module 410is then able to retrieve all the providers of the content item.

By using fingerprints, it becomes possible to identify one particularcontent item regardless of the transmission medium or the format inwhich it is encoded. For example, it can now be determined that twocontent items contain the same piece of music, even if the respectiveitems originate from different providers. Using fingerprints is alsorobust against end users tampering with information such as filename,file size or any information embedded in the content item.

The fingerprinting module 310 and the fingerprint-based identificationmodule 410 can be embodied as separate devices to be installed in theMMSC 120, or as separate devices connected to the MMSC 120. For example,the module 310 could be designed to operate as a proxy to be connectedbetween MMS application 201 and MMSC 120, making operation transparent.Similarly, the module 410 can be designed as a proxy between MMSC 120and mobile network 130. The modules 310, 410 can be embodied as onesingle device or module. They can be embodied in whole or in part inhardware or in software.

The entity operating the MMSC 120 is able to optimize content at theMMSC 120 for the receiving user agent using media mediation system 510,as shown schematically in FIG. 5. Optimization can be the adaptation ortranscoding of the format (for images e.g., GIF to JPEG), the size (forvideo e.g., from CIF to QCIF), the bitrate (for audio e.g., 128 kbps to20 kbps) and other parameters. Since this media mediation system 510 hasaccess to, and knowledge about the multimedia content of the message,the modules 310, 410 are then preferably positioned in this system 510.

An illustrative embodiment of the MMSC 120 is shown schematically inFIG. 6. The functionality of the MMSC 120 can be divided conceptuallyinto a server portion 601 and a relay portion 651. Briefly speaking, theserver portion 601 provides the store-and-forward architecture and therelay portion 651 transcodes and delivers the messages.

A message is received by the receiver component 610 of the server 601.The message is processed to detect how it should be handled and to whomit should be forwarded. In accordance with the invention, the contentsof the message is processed by the fingerprint-based identificationmodule 410 to detect whether the multimedia content of the message hasalready been forwarded. If such detection is positive, the MMSC 120takes appropriate action, as explained above.

If forwarding of the message is acceptable, there are several possiblepaths the message can take. Usually the message will be passed on to therelay 651. The relay 651 comprises the fingerprinting module 310 and themedia mediation system 510 which transcodes the message to a formsuitable for handling by the recipient. The message is then passed on tothe mobile network 130 so that it can be delivered to the intendedrecipient.

The message can also be passed on to an SMTP interface 620 connected toa network 621 such as the Internet. This interface formats themultimedia message in accordance with Internet e-mail standards, andthen forwards the message as an Internet e-mail message to the intendedrecipient. The SMTP interface 620 may or may not perform the abovefingerprint-based check on the multimedia message before forwarding it.If the SMTP interface 620 does so, further forwarding of the resultinge-mail message to another recipient via the MMS network becomesimpossible. However, as the Internet e-mail network is not very strictlyregulated, it will be very difficult to enforce forwarding restrictionsover the e-mail network itself.

If the recipient is not on the same mobile network as the sender of themultimedia message, the relay cannot forward the message itself to theintended recipient. In this case, the server passes the message on to anMMSC 630 which is connected to the correct mobile network.

The fingerprint-based identification module 410 can also be positionedin the relay portion 651 of the MMSC 120. This has the advantage thatthe module 410 and module 310 could e.g. be realized as a single device.

The functionality of the transcoding module 510 can be extended to morethan just converting between formats, sizes, bitrates etcetera Forinstance, an interface with MMS applications could be provided. Thisway, an MMS application could for instance dynamically generate contentspecifically adapted for one particular mobile device.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the appended claims. For example, the modules 310, 410could be incorporated in the user agents 101, 141. Instead of robustfingerprints, also for example cryptographic hashes, CRCs,filename-based identifiers or watermark-based identifiers could be used.A combination of different techniques for obtaining an identifier isalso possible.

The invention is of course applicable to any store-and-forward system,for example i-mode/i-mail, WAP and all future mobile premium contentways of transport, not just the MMS system described above.

The content items that can be transported could be any type of content,for example (java) games, greeting cards (still and animated), locationbased services (eg city-maps, etc.), premium information content (newsreports+pictures, stock-market info+graphics), screensavers,images/pictures (can be anything: music artists, playboy playmates,etc.), audio or video clips and so on.

It is for instance also possible to implement the fingerprinting andfingerprint-based lookup functionality according to the invention in anInternet or Intranet e-mail server system. This way e-mail messagescannot be used to indefinitely forward multimedia content. Analysis ofthe contents of e-mail attachments is already done by anti-virussoftware, and a similar approach could be used to compute fingerprintsin attachments.

In the claims, any reference signs placed between parentheses shall notbe construed as limiting the claim. The word “comprising” does notexclude the presence of elements or steps other than those listed in aclaim. The word “a” or “an” preceding an element does not exclude thepresence of a plurality of such elements. The invention can beimplemented by means of hardware comprising several distinct elements,and by means of a suitably programmed computer.

In the device claim enumerating several means, several of these meanscan be embodied by one and the same item of hardware. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage.

1. A method of controlling retransmission of a content item contained ina multimedia message, comprising receiving the message containing thecontent item from a sender together with an identifier of an intendedrecipient of the message, obtaining an identifier for the content item,retrieving approval information from a provider based on the identifierfor the content item, and controlling retransmission of the messageincluding the content item to the intended recipient based on theretrieved approval information.
 2. The method as claimed in claim 1, inwhich controlling retransmission comprises allowing retransmission ofthe message including the content item to the intended recipient andbilling the sender or the intended recipient a premium price.
 3. Themethod as claimed in claim 1, in which controlling retransmissioncomprises allowing retransmission of the message including the contentitem to the intended recipient and recording details regarding theretransmission.
 4. The method as claimed in claim 1, in whichcontrolling retransmission comprises restricting retransmission of themessage.
 5. The method as claimed in claim 4, in which restrictingretransmission comprises disallowing retransmission of the message. 6.The method as claimed in claim 4, in which restricting retransmissioncomprises allowing retransmission of the message but disallowingretransmission of the content item.
 7. The method as claimed in claim 1,in which information identifying the provider is retrieved from adatabase based on the obtained identifier.
 8. The method as claimed inclaim 7, in which the obtained identifier is stored associated with anidentifier for the sender in the database if the information identifyingthe provider could not be retrieved from the database.
 9. The method asclaimed in claim 7, in which the information identifies pluralproviders, retransmission being conditional upon approval informationretrieved from any of the plural providers.
 10. A system arranged forcontrolling retransmission of a content item contained in a multimediamessage, comprising receiving means for receiving the message containingthe content item from a sender together with an identifier of anintended recipient of the message, identifying means for obtaining anidentifier for the content item, lookup means for retrieving approvalinformation from a provider based on the identifier for the contentitem, and for signaling to conditional retransmitting means the approvalinformation, the conditional retransmitting means being arranged forcontrolling retransmission of the message including the content item tothe intended recipient based on the approval information.
 11. A mediatranscoding system, arranged for transcoding content items in amultimedia message to a format suitable for an intended recipient,comprising the system of claim
 10. 12. A computer program productarranged for causing a processor to execute the method of claim 1.